enRoute is basically taking OSGi and making it a lot easier to use than trying to start on your own with OSGi. Basically, OSGi is a fantastic foundation to build applications on, it’s like a giant concrete slab that allows you to build a hundred story building, unfortunately you have to build all the floors yourself so far and trying to get started on that is a big problem for a lot of people. What we try to do with enRoute is to focus on building a tool chain and building the libraries altogether so that you basically get started in 10 minutes and start at the fourth floor instead of having to start at the concrete slab. So, you get all the good things of OSGi, the very well written and thoroughly designed specifications, you get the capability, the requirements model, you get the enormous libraries that are out there, but you don’t have to assemble it yourself to do something simple.
2. And these are the enRoute profiles - what enRoute profiles are there?
Currently, we have the base profile, it’s all something that is still very much in development. The base profile is the minimal to get started with OSGi, so it allows you to write tutorials on OSGi and you can really focus on the incredibly powerful service model in OSGi. The next profile is going to be the debug profile, so that you can easily debug, actually that is already part of the base profile, as well, the following profile is going to be web of course, you can make web applications, single page web applications with REST, JSON RPC based, and after that we’ll go for the persistence. I started actually with the persistence, but that was a rather complicated story in Java, I found out, there was no clear, easy answers.
Yes. We do it in trivial, there is already a preliminary tutorial on GitHub about all the ideas of enRoute in there that we will work out further. Probably when this interview is going to be published it’s going to be part of the profiles as well, you have to be up and running on wizard web page, debugging it, because it’s beautiful, in the tool chain the debugging is very elegant, you save your JavaScript code or your Java code and it’s immediately available in your web browser, it’s very direct development and debugging, so you can get started in a couple of minutes and have your framework for the web application there.
Alex: Tell us about the tool chains and the tools that are involved.
Right. That was one of the key things that we saw when people were starting to use OSGi, they had to assemble all the parts together and then when you go to Ruby on Rails you see that people basically have this whole thing at once, so OSGi has beautiful scalability and the type safety and the versioning and all these powerful things for software engineering, but it didn’t really have this ease of use. So, what we do with enRoute, we select the tool chain, what we did was we picked the best tools that are currently available, I’m slightly biased there because I am one of the authors of one of them, BndTools, because those are basically the tools that are used by all people, virtually all people that build OSGi applications the way they are intended to be designed. So there is an Eclipse plugin BndTools that will be the IDE that we use. Because it’s Bnd it will run from the command line as well, so actually if you want to use Ant, you can use Maven, you can use Gradle, you can actually use the Bnd command line tool as well to build your applications. To prevent that confusion, what shall I pick, where shall I start, we start with Eclipse, we have Gradle and we have Travis as continuous integration, we put them all in one tool chain and when you start with OSGi enRoute, you basically start with this whole tool chain, which means that from out of the box you’ve got full, very nice development model in an IDE, you’ve got complete command line control using the Bnd command line utility, the Gradle as a build tool and with Travis you build continuous integration. So, of course, git is a part of that as well, every time you commit your git repository it automatically builds it and you get a mail if things fail.
That’s the beauty of BndTools, it’s kind of interesting because when we started Bnd we wrote an Eclipse plugin and there was a manual entry, build my bundle, and what Neil Bartlett did, the author of BndTools, he just took Bnd and said “well, that model sucks, let’s do it differently, let’s just continuously build it”, every time you save, the real bundle is built. If you look at PDE, which is I guess the competitor, they build also, but they are faking the real environment, they are not really doing a build step on the bundles, they are keeping the JARs in the bin directory and then when you debug they do some classpath magic to make it look like it’s a bundle. In Bnd that’s not the case, we always build a bundle, Bnd uses a macro language so you don’t repeat yourself, if there is something that you want to define, you define it in one place and you reuse it in your whole build.
So, all these cool parts of build tools, assembling resources, preprocessing resources, we do that anytime when you save, the moment you hit your Ctrl+S we rebuild the bundles and actually when you are debugging or running your application we immediately deploy them to the running OSGi framework and they are available in a quarter to half a second. I’ve never actually seen any other environment that is so fluid, there was a keynote here from Gilad Bracha on the conference where he was very proud that in the browser he could change things that immediately were available, we might not get the direct, only-in-the browser model, but we get very close to that kind of model.
The application I developed on, it’s JPM4j, is completely developing that model and there were days that I never rebooted the framework, I would just save, look, test and the edit-compile-build-debug cycle is extremely short which gives you a completely different view. And you can do that because OSGi is dynamic, we don’t have to use magic like JRebel that does all kinds of class loader magic under the covers, it’s just part of the OSGi model that when you build a bundle you can actually deploy it, restart it and when you use the OSGi service model, all those dynamics keep on working, it’s quite a beautiful development environment.
Exactly. And the beauty of that model is that because you kick the system so often, usually several times a minute, especially when you do JavaScript development which is always a little more iterative than Java development because you have to get all the names of the variables right and you don’t get good IDE support on that, at least in some of the environments, you tend to do lots of iterations, so your OSGi bundles that are running in that system as well, they get kicked all the time. So if there are any problems in your code that cannot handle that dynamic stuff they are found out rather easily, the code gets exercised and the beauty is when you deploy it you often don’t kick them not nearly as often but they get this kind of extra layer resilience because they grew up in a very hectic environment and then when they get matured later on they actually now have a stable environment. So, when things go wrong they already have this resilience to survive, because in reality bad stuff happens all the time.
Alex: So, in fact, you’re not just testing the implementation, you’re also testing the dynamism of the system as well.
Continuously, because you are continuously deploying your bundles. One of the things you are usually doing in OSGi project is you do service based programming, services require contracts to APIs. When you work in a work space you use OSGi in a single project that contains all your APIs, so you put your server interfaces in a separate project which allows you to do the governance on it rather easy, you get an architecture board, it takes a look at that.
Contracts should be stable, so you don’t change it very often, but sometimes you need to add a new API, you need to add new stuff; that project of course is the root of all dependencies because that contains all the APIs so you do a recompile of everything, every bundle gets rebuild if you change that project, it’s not a problem because it’s only API, it’s not an implementation that’s in there, but it’s a beautiful way to see what happens to your system because every interface is going to be recompiled, that bundle is going to be rebuild, all your implementations are going to be rebuilt because they need to check if they match the new API and then it’s all going to be deployed.
So you massively deploy 100 bundles in a couple of seconds, that usually takes a bit more than half a second, but it’s a beautiful exercise to see if your system keeps on working, if it can survive that and you learn because it’s very annoying as a developer that if you have a bug in your database code and you do a redeploy and then database connection gets lost and not rebuilt again, you fix it because it’s hurting me as a developer, but it creates a very robust system in the end because that code is used to being disturbed instead of what we see in a lot of monolithic applications where we build incredibly fragile trees and we go to the utmost to not disturb that tree because things will fall over and crash. OSGi embraces failure and my experience is that if you embrace failure you are so much more abused than if your try to get out of your way to make sure oh, let’s not touch it and let’s hope that things keep on working.
Alex: And presumably the contracts you were talking about get versioned as well.
Very good question, versions. OSGi started out with versions from day one, we never ignored time. If you look at almost anything we do in the society, in the real world, you’ll see that versions are not that relevant because we usually start over again, when you buy a new car it’s just a new car, when you buy a piece of software it’s kind of weird, you buy this license to get these upgrades of that, we’re building very fluid things that morph over time, it’s the same thing but it’s not really the same thing. And I think we don’t even have good words for this. If you take Maven for example, we have an artifact, but what is this thing that an artifact comes from, there is no good word for that description, what is that, a project, not really. But there is this conceptual thing that creates these artifacts over time, different versions. OSGi took versions seriously from the beginning.
If you look at Java, you see we pay an enormous price to get type safety. Java is very verbose language, to put all the type information in there, the price for this type safety is pretty steep, but type safety will only work if you can handle time in the development of a system, because over time your modules or parts of your system will change and you need to make sure that the type safety still works over time. The only way we can handle that in our world is actually versioning the modules and making sure that you only have the right versions of those modules on your classpath. Trying to handle that yourself is really, really hard because humans can’t read them, they can’t see more than three versions at the same time, we are very limited in our conceptual or cognitive abilities to work with versions. So, if you want to make this work, if you pay the price for type safety and you want to get the type safety in systems that get parts from all different kinds of places that each have their own rate of evolution, you need to have a system that manages the versions and OSGi is doing that. A couple of years ago, we had many discussions with JSR277 (Jigsaw) about versioning, fortunately the industry came, not in the Java world maybe, they also started to follow that more, but in the JavaScript world people started talking about semantic versioning where each part got a very definite meaning, which was already extremely close to what OSGi said about versioning 10 years ago, so we try to align that with the concepts. The problem is that in general the industry there is a tendency to version the artifacts, Maven artifacts have a version number and you try to assemble them together. The problem as an artifact is that it’s only part API but it also contains in big part implementation. The problem with implementations is you have a lot of transitive dependencies, so if I drag in my API I get all kinds of dependencies that I don’t actually really want to get.
So, what OSGi did – and that is one of the key discussion items when people come in new is – they say dependency should not be expressed on the artifact, dependency should be expressed on the interface, and I am not talking about the Java interface, I am talking about the surface interface which we actually define in a package because an interface is rarely an island on its own and on a surface interface you usually have helper classes, there is probably different interface for the collaboration because a service specification is actually collaboration between two parties, we call it the consumer, there is one party that consumes the service interface and there is one party that provides that interface, like when you buy a house you are the provider of that house and I am the consumer of that house.
Alex: Like JDBC, for example.
For example, yes. The nice thing about a house is that there is something nasty about the consumer and the provider that when it comes to versioning, if you’re the provider of that house, if you sell that house to me and I am the buyer of that house, then I don’t really have a problem if you put an extra room in there, if you provide a bit more, hey, that’s ok, I am fine as a consumer. The problem is if you remove a room of that house I am having a problem because I rely that the room is there, so these roles are a little bit different because if you provide a house, in software I wouldn’t even know about that extra room in there, I would never use it, so I am compatible with you, but if there is an API that says there are so many rooms in there and you don’t follow the contract and you provide the room less or the contract says there is a room extra, then you as a provider are not following the contract anymore.
So as a provider you have an extreme sensitivity to the version of the contract, any change in the contract is actually influencing the provider. But for the consumer you have much more leeway, again, there could be a lot of excess stuff in the contract, if I was compiled against an old contract I am still happy I didn’t see all the excess stuff. Actually, I see that when I explain that to people, that often it’s an insight or an aspect that they never saw that clearly, is that these two roles are incredibly important when you look at compatibility.
In OSGi we have a semantic versioning white paper that takes these different roles and we actually have annotations and you put in those annotations that this interface is implemented by the provider of the contract. If we change it we guarantee backward compatibility for the consumers, well, any change in the contract the provider has to rebuild against this contract again. Sounds a bit complicated, if you get your head around it, it was interesting when we wrote the white paper, what is this consumer and provider, why not just the implementer and the user, now you see that the people in the OSGi world are completely comfortable with it and it becomes very obvious. The cool part is we know this interface is a provider interface and that this interface is a consumer interface; for example listeners, when I listen to some events I have an interface that is implemented normally by the consumers, making a change in there is a breaking change for everybody, so you need to bump your version number to the major version number, but changing an interface for the provider is a minor because the compiler has to recompile anyway, but the consumers don’t because they are not broken by that. What we’ve done in Bnd is we’ve taken those rules, we have taken the Java language specification with the binary compatibility rules and the source compatibility rules and heuristics even and we encoded that in Bnd to calculate that. So, as I said earlier versions are incredibly complicated to human beings because our brains are not wired for that, but a computer is really, really good at that. So we look, actually we have a feature called baselining where every time you press the command save, Ctrl+S, and you go to your browser to look at your JavaScript, every time we compare your new JAR against the old JAR, we compare all the public API in there against previous public API and when there is a difference we say sorry, it’s wrong.
So, actually the interesting thing is that of you add a method to an interface in an API package, BndTools immediately shows the red squiggles and not only on that method that was added, because that one is breaking compatibility and you didn’t update your version to reflect that change, also on the places where that version is set you get also the red squiggles; and I thought I understood that and I thought I was a good guy for trying to work with that, but after we put that into Bnd and when it became available in BndTools, it scared the shit out of me how many errors I made. It is unbelievable if you see how sensitive it is that you make a change to a program and that it actually affects people outside your domain that would get killed if they see the change without the version bump.
6. The baselining, is this against the last version you compiled or the last release version?
We normally do it against the last release version, it’s of course completely settable how you want to do it. Because the release version has the final versions in it, it’s really the only one you can baseline against. Normally, with continuous build, we build continuously so when you release your same version we can work with that, we have to ignore a couple of things in the comparison to make sure that you don’t get the build dates, seeing that there is a difference, but the moment you get an API change or a resource change, you have to bump the version, that usually happens in BndTools, so there is a user that can modify those versions and then he commits it to continuous integration, then we do again the check that people follow the rules. So sometimes you bump the version two times before you release, first it’s a minor bump and then second it’s a major bump, depending on how you go. But the key thing is the developer that is making the breaking change is immediately warned that eh, not such a good idea.
What I find very often is what I usually used to do just create this extra method and later got yelled at because I broke somebody else’s code, you think oh, well, too bad, you start to think do I really need to add it here, maybe I should just make a new interface or can I get around it in another way, and I think that’s what it’s all about, you have to get the problem as early as possible so the guy causing the problem can actually solve it immediately and is aware of the problem and not somewhere down the line when somebody just before the release is going final, is doing his baseline check and then is going oh, my god, we have all these problems to solve, we’ll do it in the next release.
7. You talked about release repository, what kind of repository does BndTools publish to?
The nice thing about BndTools is that it doesn’t have a native repository, there is a plugin model for Bnd that allows you to plug in different repositories. The same is just files based repository, that is you can easily extend as well, it’s got hooks on it so you can actually connect to GitHub if you want to store your binaries on GitHub or connect it to S3 if you want to store it on S3. Then we have the capabilities requirements repositories, there is an OSGi model based on capabilities and requirements that allows to not only handle the OSGi metadata dependencies, it also allows you to have third party dependencies which could range from the needs of your certain certificates. That standard we use to create repositories that take artifacts and describe it with these requirements and capabilities and then allows us to resolve all the dependencies if needed and then the most intriguing one I think at the moment is JPM4j which I developed in my sabbatical two years ago.
JPM4j is a complete index of Maven Central, but it is more because it also has repositories which means you can use it to release your products into and then a couple of others, the enterprise bundle repository and anybody that can point me to a place where I can find JARs and notify me when they change, get automatically indexed, so JPM4j is basically 99% maven because there are currently 600,000 JARs in there organized in 100,000 different programs, it has quadrupled in one year, Maven Central is on an exponential curve at the moment, I guess for the camera it would be like this, it’s amazing. The interesting thing is that because JPM4j is not trying to be pedantic and to put a namespace on things, the native namespace of JPM4j is SHAs, I don’t care what namespace people are using, OSGi bundles symbolic names, Maven artifact group IDE, I don’t care, I collect all the information, put it in an index, I don’t store it myself, it stores the URLs, each artifact can have multiple URLs, and that repository is connected, we also made a plugin for Bnd, which means that in BndTools I can nicely create a repository out of the whole Maven Central. One of the key things we found rather early is that you have to be rather careful with repositories if you start to go to capabilities and requirements, when you start to go resolving.
If you look at Maven, one of the best practices is don’t use version rangers, they are part of the Maven system. The reason you shouldn’t use version rangers is that once you start using version rangers you get a dependency on the constituents in the repository. And what I mean by that today I resolve that version range which means it gives me choice, it goes to the repository and sees there are six versions, and let’s say it picks the latest, so today I compile against 2.1, tomorrow somebody inserts 2.2 and now I compile and resolves oh, I compile against 2.2, there might be a bug introduced in 2.2 or an incompatibility, so suddenly my build that was working fine yesterday is not working anymore while nothing, at least in my opinion, was changed. So once you start realizing that you get these really powerful tools resolving and allowing the computer to help you build classpaths, they have an extreme dependency on what’s happening in the outside world and in the world where I come from, in the embedded world, is we very often have this requirement that we need to be able to build our products ten years from now.
If you have a problem, Maven Central is going to look very different ten years from now, I’m not saying it’s going to be anything removed but I’m pretty sure it’s going to have 2 or 3 million or even more JARs in there organized in hundreds of thousands of programs, I cannot rely on my program compiling correctly and working correctly if it’s having to work against such a completely different beast at the time. Realizing that is what we do on the BndTools side is realizing that whenever the computer does something without a user present, like an offline build, a continuous integration build, or production or sending into testing, when there is no human that can make a normal common sense decision and things change, you want to have as much determinism as possible, you don’t want any randomness in your process. So you want to get the flexibility in the design phase, you want to have design tools to allow you to design classpath out of all kinds of components where you really want to know that everything is compatible, that things are the right versions, is the design phase, there you can have a user who can make the choice.
So what we do in Bnd is that instead of directly going in the repository, we have a text file that describes the view on the repository and we have a very nice editor in BndTools when you want to look for an artifact, I’m looking for the Jackson library, I can search in the repository local, if it’s not there, there’s a link, it starts in the JVM window in BndTools, in Eclipse, it’s inside Eclipse, it looks like it’s a plugin and you can actually drag the version in your classpath tab or you can actually drop it on the repository. It changes the text file which is completely on the version control, so if you checked out that project five years from now you will see the identical repository it’s saw now. So the dependency is gone because we’ve got the SHAs of the artifacts, so even if somebody puts the same name in there for a different thing, we will get exactly the identical artifact. So we strive to get a system where you get bit-wise identical outputs whenever you build it, and only when there is a user there we allow you to change the configuration, verify the configuration, support you to not have to change all these different things, visualize the consequences of the decisions you make, freeze it so that later on everything is going to work as you’d expect it to work.
Yes.
Alex: And then the assets can be sourced from anywhere that hosts the same binary content.
One of the thing that you need to do, especially for the embedded market, is not even rely whether that Maven Central is going to be out there, because we have the text file contains all the details, we can create a big zip file that can contain all the dependencies that are at a certain moment in time, so when you do an official release you can say ok, now I want to freeze the whole thing, but the zip file can act as a repository for that as well. The key thing is what you compile against, what your environment looks like is there is no user everything is defined up to the bit, there is no uncertainty, there is no let me look there and see if I can find it and it’s not there and let me look here and see if I can find it, oh, maybe it’s over there, there’s no uncertainty, it’s fully deterministic what you built.
So, these two things are really important because it’s the usability to have design tools that have all the flexibility to allow you to see the consequences of your decisions which you’d be very flexible and that where OSGi shines because we minimize the coupling, you have so many choices to compose things and to assemble things together, but once you’ve made that decision as a designer, as a developer, you don’t want a surprise on your continuous integration because something over there changed, that’s where most things go wrong, there are these assumptions that it’s going to be the same, oh, it works on my system, yes, so what, it has to work on continuous integration as well, you don’t have anything on the file system, it starts with a completely clean file system, you want to get the bit-wise identical results on my continuous integration as when I get on my BndTools when I do it offline. I don’t want to change these horrible to diagnose problems, why is it different over there, it should be identical.
We made a very conscious decision, I think because we looked at Java and I think one the main problems with Java is that Java developers have a very serious problem with committing themselves, there is this famous blog from Joel I think about software factory factory, factory of factories.
Alex: That’s Steve Yegge’s post about the Kingdom of Nouns.
Could be. It was a really interesting blog post about how we tend to abstract and move things out of our way which is really cool because there is no computer problem that you cannot solve by adding an extra level of interaction, so don’t get me wrong, I am doing it all the time. But if you are a newcomer it’s bewildering, you just want to parse an XML document, you go to JAXB, DocumentBuilder, Xpath, the amount of choices is bewildering. So what we did with enRoute was we created a single chain, no optionality but the beauty of OSGi is of course that is incredibly modular, and the beauty about modularity is that you try to know as little as possible, you try to be explicit in what you assume, but you try to minimize the assumptions as much as possible and stay local as much as possible. So the idea is that we create from the enRoute project we create a single chain, but we allow other people to insert alternatives.
Now, you mentioned NetBeans and IntelliJ, there is a bug on IntelliJ, I don’t what the state of that is at the moment, they promised that they would start working on integrating Bnd there as well, because BndTools is getting some traction at the moment, people see it as a very nice development environment, not just for OSGi, and the baselining of course and some of the core ideas are attractive for enterprise developers. We’d love to work with the IntelliJ guys to get Bnd integrated in it, I sometimes call Bnd the little engine that built, Bndlib is a library that has no dependencies, except for the Java VM, it runs on Java 5, if you’ve got that, then Bndlib runs. Bndlib provides you complete integration, if you connect Bndlib in IntelliJ or in NetBeans or whatever environment you’ve got, you can do anything that BndTools can do. Of course, BndTools is doing the user interface, it has all the drag and drop, the fancy stuff for people to call up menus, but on the command line you can do exactly the same, every feature in BndTools is reflected with an API call to Bndlib where the actual low level grunt work is done.
So, we’d love to work with anybody that says we want to integrate that, but for the enRoute project we will have a primary chain that is complete, end-to-end, so that as a beginner you are not faced with all the choices, should I use and or Gradle. It’s trivial for us to add Ant, actually we have a Maven plugin at the moment, that’s not the Maven bundle plugin that quite a lot of people are using (it’s actually one of the most popular Maven plugins on Maven Central, except for the Sonatype plugins). There is a new Maven plugin that actually allows you to build a Bnd project, so you make with Bndtools, you can add a Pom to it, and then you can use Maven to build that Bnd project, it will not use the Maven repository model, it will use the Bnd repository model, because we need to create the same bits as we do in Bndtools and we do with other tools. So, Bndlib is completely uncoupled, you can put it anywhere and the moment you put it anywhere you have all the features of the Bnd model available to you.
Alex: So, if you’re working in a Maven only environment or editing it as a Maven project in IntelliJ, it will just work out of the box.
Yes. The problem is you don’t get some of the really nice features on the usability level, but yes. And I hope IntelliJ gets interested enough to work with us to basically provide the kind of BndTools experience, the user interface aspects and combine it with their really good editors so that you have the best of both worlds. I’d love to work with them on that, but for enRoute we took our single, because we don’t want to have optionality, it should be clear from the beginning, but of course, you can once IntelliJ would take Bndlib and adopt it and put it in their base, you don’t need to use Eclipse, again because of the underline model is Bndlib, anybody who has that will be able to collaborate in this game.
10. Where can people find out more information from enRoute and download the tool chain setup?
At the time this video is there, there is going to be a link enRoute, so that’s the obvious URL and you’ll find information about it on the home page of OSGi, as well.
Alex: Peter Kriens, thank you very much.
You’re welcome.